True Asymptotic Natural Gradient Optimization

We introduce a simple algorithm, True Asymptotic Natural Gradient Optimization (TANGO), that converges to a true natural gradient descent in the limit of small learning rates, without explicit Fisher matrix estimation. For quadratic models the algorithm is also an instance of averaged stochastic gradient, where the parameter is a moving average of a “fast”, constant-rate gradient descent. TANGO appears as a particular de-linearization of averaged SGD, and is sometimes quite different on non-quadratic models. This further connects averaged SGD and natural gradient, both of which are arguably optimal asymptotically. In large dimension, small learning rates will be required to approximate the natural gradient well. Still, this shows it is possible to get arbitrarily close to exact natural gradient descent with a lightweight algorithm. Let pθ(y|x) be a probabilistic model for predicting output values y from inputs x (x = ∅ for unsupervised learning). Consider the associated log-loss l(y|x) := − ln pθ(y|x) (1) Given a dataset D of pairs (x, y), we optimize the average log-loss over θ via a momentum-like gradient descent. Definition 1 (TANGO). Let δtk 6 1 be a sequence of learning rates and let γ > 0. Set v0 = 0. Iterate the following: • Select a sample (xk, yk) at random in the dataset D. • Generate a pseudo-sample ỹk for input xk according to the predictions of the current model, ỹk ∼ pθ(ỹk|xk) (or just ỹk = yk for the “outer product” variant). Compute gradients gk ← ∂l(yk|xk) ∂θ , g̃k ← ∂l(ỹk|xk) ∂θ (2) • Update the velocity and parameter via vk = (1− δtk−1)vk−1 + γgk − γ(1− δtk−1)(v⊤ k−1 g̃k)g̃k (3) θk = θk−1 − δtkvk (4)